MPMQA: Multimodal Question Answering on Product Manuals

Visual contents, such as illustrations and images, play a big role in product manual understanding. Existing Product Manual Question Answering (PMQA) datasets tend to ignore visual contents only retain textual parts. In this work, emphasize the importance of multimodal we propose Multimodal (MPMQA) task. For each question, MPMQA requires model not process but also provide answers. To support MPMQA, large-scale dataset PM209 is constructed with human annotations, which contains 209 manuals from 27 well-known consumer electronic brands. Human annotations include 6 types semantic regions for 22,021 pairs question answer. Especially, answer consists sentence related manuals. Taking into account length fact that always small number pages, can be naturally split two subtasks: retrieving most pages then generating We further unified perform these subtasks all together achieve comparable performance multiple task-specific models. The available at https://github.com/AIM3-RUC/MPMQA.